{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Hyperparameter tuning with Ray Tune\n",
    "\n",
    "Hyperparameter tuning can make the difference between an average model and a highly\n",
    "accurate one. Often simple things like choosing a different learning rate or changing\n",
    "a network layer size can have a dramatic impact on your model performance.\n",
    "\n",
    "Fortunately, there are tools that help with finding the best combination of parameters.\n",
    "[Ray Tune](https://docs.ray.io/en/latest/tune.html) is an industry standard tool for\n",
    "distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search\n",
    "algorithms, integrates with TensorBoard and other analysis libraries, and natively\n",
    "supports distributed training through [Ray's distributed machine learning engine](https://ray.io/).\n",
    "\n",
    "In this tutorial, we will show you how to integrate Ray Tune into your PyTorch\n",
    "training workflow. We will extend [this tutorial from the PyTorch documentation](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html) for training\n",
    "a CIFAR10 image classifier.\n",
    "\n",
    "As you will see, we only need to add some slight modifications. In particular, we\n",
    "need to\n",
    "\n",
    "1. wrap data loading and training in functions,\n",
    "2. make some network parameters configurable,\n",
    "3. add checkpointing (optional),\n",
    "4. and define the search space for the model tuning\n",
    "\n",
    "|\n",
    "\n",
    "To run this tutorial, please make sure the following packages are\n",
    "installed:\n",
    "\n",
    "-  ``ray[tune]``: Distributed hyperparameter tuning library\n",
    "-  ``torchvision``: For the data transformers\n",
    "\n",
    "## Setup / Imports\n",
    "Let's start with the imports:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "import numpy as np\n",
    "import os\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import random_split\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "from ray import tune\n",
    "from ray.tune import CLIReporter\n",
    "from ray.tune.schedulers import ASHAScheduler\n",
    "\n",
    "import torch.cuda\n",
    "from DataLoad.DataLoad import SetSplit, LoadNirtest\n",
    "from Preprocessing.Preprocessing import Preprocessing\n",
    "from WaveSelect.WaveSelcet import SpctrumFeatureSelcet\n",
    "from Regression.Rgs import QuantitativeAnalysis\n",
    "import multiprocessing\n",
    "import programParameter.stringConfig as STRING_CONFIG\n",
    "import programParameter.threadConfig as THREAD_CONFIG\n",
    "from Regression.CNN import ZspPocessnew\n",
    "from Evaluate.RgsEvaluate import ModelRgsevaluatePro\n",
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the imports are needed for building the PyTorch model. Only the last three\n",
    "imports are for Ray Tune.\n",
    "\n",
    "## Data loaders\n",
    "We wrap the data loaders in their own function and pass a global data directory.\n",
    "This way we can share a data directory between different trials.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# def load_data(data_dir=\"./data\"):\n",
    "#     transform = transforms.Compose([\n",
    "#         transforms.ToTensor(),\n",
    "#         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "#     ])\n",
    "#\n",
    "#     trainset = torchvision.datasets.CIFAR10(\n",
    "#         root=data_dir, train=True, download=True, transform=transform)\n",
    "#\n",
    "#     testset = torchvision.datasets.CIFAR10(\n",
    "#         root=data_dir, train=False, download=True, transform=transform)\n",
    "#\n",
    "#     return trainset, testset\n",
    "\n",
    "def load_data(data, label, ProcessMethods, FslecetedMethods, SetSplitMethods):\n",
    "    \"\"\"\n",
    "    这个函数主要服务于Ray Tune流水线\n",
    "    :param data:\n",
    "    :param label:\n",
    "    :param ProcessMethods:\n",
    "    :param FslecetedMethods:\n",
    "    :param SetSplitMethods:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    processedData = Preprocessing(ProcessMethods, data)\n",
    "    featureData, labels = SpctrumFeatureSelcet(FslecetedMethods, processedData, label)\n",
    "    X_train, X_test, y_train, y_test = SetSplit(SetSplitMethods, featureData, labels, test_size=0.2, randomseed=123)\n",
    "    return X_train, X_test, y_train, y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configurable neural network\n",
    "We can only tune those parameters that are configurable. In this example, we can specify\n",
    "the layer sizes of the fully connected layers:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# class Net(nn.Module):\n",
    "#     def __init__(self, l1=120, l2=84):\n",
    "#         super(Net, self).__init__()\n",
    "#         self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "#         self.pool = nn.MaxPool2d(2, 2)\n",
    "#         self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "#         self.fc1 = nn.Linear(16 * 5 * 5, l1)\n",
    "#         self.fc2 = nn.Linear(l1, l2)\n",
    "#         self.fc3 = nn.Linear(l2, 10)\n",
    "#\n",
    "#     def forward(self, x):\n",
    "#         x = self.pool(F.relu(self.conv1(x)))\n",
    "#         x = self.pool(F.relu(self.conv2(x)))\n",
    "#         x = x.view(-1, 16 * 5 * 5)\n",
    "#         x = F.relu(self.fc1(x))\n",
    "#         x = F.relu(self.fc2(x))\n",
    "#         x = self.fc3(x)\n",
    "#         return x\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self, acti_func, cnn_depth):\n",
    "        super(Net, self).__init__()\n",
    "        self.layers = nn.ModuleList([])\n",
    "        input_channel = 1\n",
    "        output_channel = 16\n",
    "        for i in range(1, cnn_depth):\n",
    "            self.layers.append(nn.Conv1d(input_channel, output_channel, 3, padding=1))\n",
    "            self.layers.append(nn.BatchNorm1d(num_features=output_channel))\n",
    "            self.layers.append(ac_dict[acti_func](inplace=True))\n",
    "            self.layers.append(nn.MaxPool1d(2, 2))\n",
    "            input_channel = output_channel\n",
    "            output_channel = output_channel * 2\n",
    "        # linear[c_num-1]\n",
    "        self.reg = nn.Sequential(\n",
    "            nn.Linear(5120, 1000),  # 根据自己数据集修改\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(1000, 500),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(500, 1), )\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = x\n",
    "        for layer in self.layers:\n",
    "            out = layer(out)\n",
    "        out = out.flatten(start_dim=1)\n",
    "        # out = out.view(-1,self.output_channel)\n",
    "        out = self.reg(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The train function\n",
    "Now it gets interesting, because we introduce some changes to the example [from the PyTorch\n",
    "documentation](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html).\n",
    "\n",
    "We wrap the training script in a function ``train_cifar(config, checkpoint_dir=None, data_dir=None)``.\n",
    "As you can guess, the ``config`` parameter will receive the hyperparameters we would like to\n",
    "train with. The ``checkpoint_dir`` parameter is used to restore checkpoints. The ``data_dir`` specifies\n",
    "the directory where we load and store the data, so multiple runs can share the same data source.\n",
    "\n",
    "```python\n",
    "net = Net(config[\"l1\"], config[\"l2\"])\n",
    "\n",
    "if checkpoint_dir:\n",
    "    model_state, optimizer_state = torch.load(\n",
    "        os.path.join(checkpoint_dir, \"checkpoint\"))\n",
    "    net.load_state_dict(model_state)\n",
    "    optimizer.load_state_dict(optimizer_state)\n",
    "```\n",
    "The learning rate of the optimizer is made configurable, too:\n",
    "\n",
    "```python\n",
    "optimizer = optim.SGD(net.parameters(), lr=config[\"lr\"], momentum=0.9)\n",
    "```\n",
    "We also split the training data into a training and validation subset. We thus train on\n",
    "80% of the data and calculate the validation loss on the remaining 20%. The batch sizes\n",
    "with which we iterate through the training and test sets are configurable as well.\n",
    "\n",
    "### Adding (multi) GPU support with DataParallel\n",
    "Image classification benefits largely from GPUs. Luckily, we can continue to use\n",
    "PyTorch's abstractions in Ray Tune. Thus, we can wrap our model in ``nn.DataParallel``\n",
    "to support data parallel training on multiple GPUs:\n",
    "\n",
    "```python\n",
    "device = \"cpu\"\n",
    "if torch.cuda.is_available():\n",
    "    device = \"cuda:0\"\n",
    "    if torch.cuda.device_count() > 1:\n",
    "        net = nn.DataParallel(net)\n",
    "net.to(device)\n",
    "```\n",
    "By using a ``device`` variable we make sure that training also works when we have\n",
    "no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly,\n",
    "like this:\n",
    "\n",
    "```python\n",
    "for i, data in enumerate(trainloader, 0):\n",
    "    inputs, labels = data\n",
    "    inputs, labels = inputs.to(device), labels.to(device)\n",
    "```\n",
    "The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray\n",
    "also supports [fractional GPUs](https://docs.ray.io/en/master/using-ray-with-gpus.html#fractional-gpus)\n",
    "so we can share GPUs among trials, as long as the model still fits on the GPU memory. We'll come back\n",
    "to that later.\n",
    "\n",
    "### Communicating with Ray Tune\n",
    "\n",
    "The most interesting part is the communication with Ray Tune:\n",
    "\n",
    "```python\n",
    "with tune.checkpoint_dir(epoch) as checkpoint_dir:\n",
    "    path = os.path.join(checkpoint_dir, \"checkpoint\")\n",
    "    torch.save((net.state_dict(), optimizer.state_dict()), path)\n",
    "\n",
    "tune.report(loss=(val_loss / val_steps), accuracy=correct / total)\n",
    "```\n",
    "Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically,\n",
    "we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics\n",
    "to decide which hyperparameter configuration lead to the best results. These metrics\n",
    "can also be used to stop bad performing trials early in order to avoid wasting\n",
    "resources on those trials.\n",
    "\n",
    "The checkpoint saving is optional, however, it is necessary if we wanted to use advanced\n",
    "schedulers like\n",
    "[Population Based Training](https://docs.ray.io/en/master/tune/tutorials/tune-advanced-tutorial.html).\n",
    "Also, by saving the checkpoint we can later load the trained models and validate them\n",
    "on a test set.\n",
    "\n",
    "### Full training function\n",
    "\n",
    "The full code example looks like this:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# def train_cifar(config, checkpoint_dir=None, data_dir=None):\n",
    "#     # 初始化模型\n",
    "#     net = Net(config[\"l1\"], config[\"l2\"])\n",
    "#\n",
    "#     # 设置运行设备\n",
    "#     device = \"cpu\"\n",
    "#     if torch.cuda.is_available():\n",
    "#         device = \"cuda:0\"\n",
    "#         if torch.cuda.device_count() > 1:\n",
    "#             net = nn.DataParallel(net)\n",
    "#     net.to(device)\n",
    "#\n",
    "#     # 设置 损失函数\n",
    "#     criterion = nn.CrossEntropyLoss()\n",
    "#\n",
    "#     # 设置 模型优化器\n",
    "#     optimizer = optim.SGD(net.parameters(), lr=config[\"lr\"], momentum=0.9)\n",
    "#\n",
    "#     # 多次训练之间共享数据\n",
    "#     if checkpoint_dir:\n",
    "#         model_state, optimizer_state = torch.load(\n",
    "#             os.path.join(checkpoint_dir, \"checkpoint\"))\n",
    "#         net.load_state_dict(model_state)\n",
    "#         optimizer.load_state_dict(optimizer_state)\n",
    "#\n",
    "#     # 获得训练和测试数据集合\n",
    "#     trainset, testset = load_data(data_dir)\n",
    "#\n",
    "#     test_abs = int(len(trainset) * 0.8)\n",
    "#     # 数据集划分\n",
    "#     train_subset, val_subset = random_split(\n",
    "#         trainset, [test_abs, len(trainset) - test_abs])\n",
    "#\n",
    "#     trainloader = torch.utils.data.DataLoader(\n",
    "#         train_subset,\n",
    "#         batch_size=int(config[\"batch_size\"]),\n",
    "#         shuffle=True,\n",
    "#         num_workers=8)\n",
    "#     valloader = torch.utils.data.DataLoader(\n",
    "#         val_subset,\n",
    "#         batch_size=int(config[\"batch_size\"]),\n",
    "#         shuffle=True,\n",
    "#         num_workers=8)\n",
    "#\n",
    "#     # 开始训练\n",
    "#     for epoch in range(10):  # loop over the dataset multiple times\n",
    "#         running_loss = 0.0\n",
    "#         epoch_steps = 0\n",
    "#         for i, data in enumerate(trainloader, 0):\n",
    "#             # get the inputs; data is a list of [inputs, labels]\n",
    "#             inputs, labels = data\n",
    "#             inputs, labels = inputs.to(device), labels.to(device)\n",
    "#\n",
    "#             # zero the parameter gradients\n",
    "#             optimizer.zero_grad()\n",
    "#\n",
    "#             # forward + backward + optimize\n",
    "#             outputs = net(inputs)\n",
    "#             loss = criterion(outputs, labels)\n",
    "#             loss.backward()\n",
    "#             optimizer.step()\n",
    "#\n",
    "#             # print statistics\n",
    "#             running_loss += loss.item()\n",
    "#             epoch_steps += 1\n",
    "#             if i % 2000 == 1999:  # print every 2000 mini-batches\n",
    "#                 print(\"[%d, %5d] loss: %.3f\" % (epoch + 1, i + 1,\n",
    "#                                                 running_loss / epoch_steps))\n",
    "#                 running_loss = 0.0\n",
    "#\n",
    "#         # Validation loss\n",
    "#         val_loss = 0.0\n",
    "#         val_steps = 0\n",
    "#         total = 0\n",
    "#         correct = 0\n",
    "#         for i, data in enumerate(valloader, 0):\n",
    "#             with torch.no_grad():\n",
    "#                 inputs, labels = data\n",
    "#                 inputs, labels = inputs.to(device), labels.to(device)\n",
    "#\n",
    "#                 outputs = net(inputs)\n",
    "#                 _, predicted = torch.max(outputs.data, 1)\n",
    "#                 total += labels.size(0)\n",
    "#                 correct += (predicted == labels).sum().item()\n",
    "#\n",
    "#                 loss = criterion(outputs, labels)\n",
    "#                 val_loss += loss.cpu().numpy()\n",
    "#                 val_steps += 1\n",
    "#\n",
    "#         with tune.checkpoint_dir(epoch) as checkpoint_dir:\n",
    "#             path = os.path.join(checkpoint_dir, \"checkpoint\")\n",
    "#             torch.save((net.state_dict(), optimizer.state_dict()), path)\n",
    "#\n",
    "#         tune.report(loss=(val_loss / val_steps), accuracy=correct / total)\n",
    "#     print(\"Finished Training\")\n",
    "\n",
    "def train(config, checkpoint_dir=None, data_dir=None, loss_func=nn.MSELoss(), data_wrapUp=None):\n",
    "    # 实例化模型 （自动超参：激活函数、网络深度）\n",
    "    network = Net(config[STRING_CONFIG.acti_func], config[STRING_CONFIG.cnn_depth])\n",
    "\n",
    "    # 设置运行设备\n",
    "    device = \"cpu\"\n",
    "    if torch.cuda.is_available():\n",
    "        device = \"cuda:0\"\n",
    "        if torch.cuda.device_count() > 1:\n",
    "            network = nn.DataParallel(network)\n",
    "    network.to(device)\n",
    "\n",
    "    # 设置 损失函数\n",
    "    criterion = loss_func\n",
    "\n",
    "    # 使用字典映射调用优化器\n",
    "    OPTIM_DICT = {\n",
    "        STRING_CONFIG.Adam: torch.optim.Adam(network.parameters(), lr=config[STRING_CONFIG.lr], weight_decay=0.001),\n",
    "        STRING_CONFIG.SGD: torch.optim.SGD(network.parameters(), lr=config[STRING_CONFIG.lr],\n",
    "                                           momentum=config[STRING_CONFIG.momentum]),\n",
    "        # 接下来的激活函数 torch并不支持。\n",
    "        # 'Adagrad': torch.optim.Adagrad(network.parameters(), lr=0.01),\n",
    "        STRING_CONFIG.Adadelta: torch.optim.Adadelta(network.parameters()),\n",
    "        STRING_CONFIG.RMSprop: torch.optim.RMSprop(network.parameters(), lr=config[STRING_CONFIG.lr], alpha=0.99),\n",
    "        STRING_CONFIG.Adamax: torch.optim.Adamax(network.parameters(), lr=config[STRING_CONFIG.lr], betas=(0.9, 0.999)),\n",
    "        # 'LBFGS': torch.optim.LBFGS(network.parameters(), lr=0.01)\n",
    "    }\n",
    "    # 设置 模型优化器 (自动超参：模型优化器)\n",
    "    optimizer = OPTIM_DICT[config[STRING_CONFIG.optimizer]]\n",
    "    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer,\n",
    "                                                           mode=\"min\",\n",
    "                                                           factor=0.5, verbose=1,\n",
    "                                                           eps=1e-06,\n",
    "                                                           patience=20)\n",
    "\n",
    "    # 多次训练之间共享数据\n",
    "    if checkpoint_dir:\n",
    "        model_state, optimizer_state = torch.load(\n",
    "            os.path.join(checkpoint_dir, \"checkpoint\"))\n",
    "        net.load_state_dict(model_state)\n",
    "        optimizer.load_state_dict(optimizer_state)\n",
    "\n",
    "    # 获得（划分处理完成后的）训练和测试数据集合\n",
    "    X_train, X_test, y_train, y_test = data_wrapUp\n",
    "\n",
    "    # 数据集预处理以及数据集载入\n",
    "    trainSet, testSet, yscaler = ZspPocessnew(X_train, X_test, y_train, y_test, need=True)\n",
    "\n",
    "    test_abs = int(len(trainSet) * 0.8)\n",
    "    # 数据集划分\n",
    "    train_subset, val_subset = random_split(trainSet, [test_abs, len(trainSet) - test_abs])\n",
    "    train_loader = torch.utils.data.DataLoader(dataset=train_subset,\n",
    "                                               batch_size=BATCH_SIZE,\n",
    "                                               shuffle=True,\n",
    "                                               num_workers=THREAD_CONFIG.num_workers)\n",
    "    val_loader = torch.utils.data.DataLoader(dataset=val_subset,\n",
    "                                             batch_size=TBATCH_SIZE,\n",
    "                                             shuffle=True,\n",
    "                                             num_workers=THREAD_CONFIG.num_workers)\n",
    "\n",
    "    print(\"Start Training!\\n\")\n",
    "    # 以 per epoch 为单位记录模型训练损失 & 以 per iteration 为单位记录模型训练损失\n",
    "    trainIter_losses = []\n",
    "    epoch_loss = []\n",
    "    for epoch in range(config[STRING_CONFIG.epochs]):\n",
    "        train_rmse = []\n",
    "        train_r2 = []\n",
    "        train_mae = []\n",
    "        avg_loss = []\n",
    "\n",
    "        ################### 记录以epoch来记录 loss ###################\n",
    "        temp_trainLosses = 0\n",
    "        ################### 记录以epoch来记录 loss ###################\n",
    "\n",
    "        running_loss = 0.0\n",
    "        epoch_steps = 0\n",
    "        for i, data in enumerate(train_loader, 0):\n",
    "            # 获得输入，data is a list of [inputs, labels]\n",
    "            inputs, labels = data  # 输入和标签都等于data\n",
    "            inputs = Variable(inputs).type(torch.FloatTensor).to(device)  # batch x\n",
    "            labels = Variable(labels).type(torch.FloatTensor).to(device)  # batch y\n",
    "\n",
    "            # zero the parameter gradients\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # forward + backward + optimize\n",
    "            outputs = network(inputs)\n",
    "            loss = criterion(outputs, labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            pred = output.detach().cpu().numpy()\n",
    "            y_true = labels.detach().cpu().numpy()\n",
    "            trainIter_losses.append(loss.item())\n",
    "            temp_trainLosses = loss.item()\n",
    "            rmse, R2, mae = ModelRgsevaluatePro(pred, y_true, yscaler)\n",
    "            avg_train_loss = np.mean(trainIter_losses)\n",
    "            train_rmse.append(rmse)\n",
    "            train_r2.append(R2)\n",
    "            train_mae.append(mae)\n",
    "\n",
    "            # print statistics\n",
    "            running_loss += loss.item()\n",
    "            epoch_steps += 1\n",
    "            if i % 2000 == 1999:  # print every 2000 mini-batches\n",
    "                print(\"[%d, %5d] loss: %.3f\" % (epoch + 1, i + 1,\n",
    "                                                running_loss / epoch_steps))\n",
    "                running_loss = 0.0\n",
    "\n",
    "        epoch_loss.append(temp_trainLosses)\n",
    "        avgrmse = np.mean(train_rmse)\n",
    "        avgr2 = np.mean(train_r2)\n",
    "        avgmae = np.mean(train_mae)\n",
    "        # print('Epoch:{}, TRAIN:rmse:{}, R2:{}, mae:{}'.format((epoch + 1), (avgrmse), (avgr2), (avgmae)))\n",
    "        # print('lr:{}, avg_train_loss:{}'.format((optimizer.param_groups[0]['lr']), avg_train_loss))\n",
    "\n",
    "        avg_loss.append(np.array(avg_train_loss))\n",
    "\n",
    "        # Validation loss\n",
    "        val_loss = 0.0\n",
    "        val_steps = 0\n",
    "        total = 0\n",
    "        correct = 0\n",
    "\n",
    "        test_rmse = []\n",
    "        test_r2 = []\n",
    "        test_mae = []\n",
    "        for i, data in enumerate(val_loader, 0):\n",
    "            with torch.no_grad():\n",
    "                inputs, labels = data  # 输入和标签都等于data\n",
    "                inputs = Variable(inputs).type(torch.FloatTensor).to(device)  # batch x\n",
    "                labels = Variable(labels).type(torch.FloatTensor).to(device)  # batch y\n",
    "\n",
    "                outputs = network(inputs)\n",
    "                pred = outputs.detach().cpu().numpy()\n",
    "                # y_pred.append(pred.astype(int))\n",
    "                y_true = labels.detach().cpu().numpy()\n",
    "                # y.append(y_true.astype(int))\n",
    "                rmse, R2, mae = ModelRgsevaluatePro(pred, y_true, yscaler)\n",
    "                test_rmse.append(rmse)\n",
    "                test_r2.append(R2)\n",
    "                test_mae.append(mae)\n",
    "\n",
    "                _, predicted = torch.max(outputs.data, 1)\n",
    "                total += labels.size(0)\n",
    "                correct += (predicted == labels).sum().item()\n",
    "\n",
    "                loss = criterion(outputs, labels)\n",
    "                val_loss += loss.cpu().numpy()\n",
    "                val_steps += 1\n",
    "\n",
    "            avgrmse = np.mean(test_rmse)\n",
    "            avgr2 = np.mean(test_r2)\n",
    "            avgmae = np.mean(test_mae)\n",
    "            # print('EPOCH：{}, TEST: rmse:{}, R2:{}, mae:{}\\n'.format((epoch + 1), (avgrmse), (avgr2), (avgmae)))\n",
    "            # 将每次测试结果实时写入acc.txt文件中\n",
    "            scheduler.step(rmse)\n",
    "\n",
    "        with tune.checkpoint_dir(epoch) as checkpoint_dir:\n",
    "            path = os.path.join(checkpoint_dir, \"checkpoint\")\n",
    "            torch.save((net.state_dict(), optimizer.state_dict()), path)\n",
    "\n",
    "        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)\n",
    "    print(\"Finished Training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, most of the code is adapted directly from the original example.\n",
    "\n",
    "## Test set accuracy\n",
    "Commonly the performance of a machine learning model is tested on a hold-out test\n",
    "set with data that has not been used for training the model. We also wrap this in a\n",
    "function:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# def test_accuracy(net, device=\"cpu\"):\n",
    "#     trainset, testset = load_data()\n",
    "#\n",
    "#     testloader = torch.utils.data.DataLoader(\n",
    "#         testset, batch_size=4, shuffle=False, num_workers=2)\n",
    "#\n",
    "#     correct = 0\n",
    "#     total = 0\n",
    "#     with torch.no_grad():\n",
    "#         for data in testloader:\n",
    "#             images, labels = data\n",
    "#             images, labels = images.to(device), labels.to(device)\n",
    "#             outputs = net(images)\n",
    "#             _, predicted = torch.max(outputs.data, 1)\n",
    "#             total += labels.size(0)\n",
    "#             correct += (predicted == labels).sum().item()\n",
    "#\n",
    "#     return correct / total\n",
    "\n",
    "def test_accuracy(net, device=\"cpu\", data_wrapUp=None):\n",
    "    X_train, X_test, y_train, y_test = data_wrapUp\n",
    "\n",
    "    # 数据集预处理以及数据集载入\n",
    "    trainSet, testSet, yscaler = ZspPocessnew(X_train, X_test, y_train, y_test, need=True)\n",
    "\n",
    "    testloader = torch.utils.data.DataLoader(\n",
    "        testset, batch_size=4, shuffle=False, num_workers=2)\n",
    "\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    with torch.no_grad():\n",
    "        for data in testloader:\n",
    "            images, labels = data\n",
    "            images, labels = images.to(device), labels.to(device)\n",
    "            outputs = net(images)\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            total += labels.size(0)\n",
    "            correct += (predicted == labels).sum().item()\n",
    "\n",
    "    return correct / total\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function also expects a ``device`` parameter, so we can do the\n",
    "test set validation on a GPU.\n",
    "\n",
    "## Configuring the search space\n",
    "Lastly, we need to define Ray Tune's search space. Here is an example:\n",
    "\n",
    "```python\n",
    "config = {\n",
    "    \"l1\": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),\n",
    "    \"l2\": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),\n",
    "    \"lr\": tune.loguniform(1e-4, 1e-1),\n",
    "    \"batch_size\": tune.choice([2, 4, 8, 16])\n",
    "}\n",
    "```\n",
    "The ``tune.sample_from()`` function makes it possible to define your own sample\n",
    "methods to obtain hyperparameters. In this example, the ``l1`` and ``l2`` parameters\n",
    "should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256.\n",
    "The ``lr`` (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly,\n",
    "the batch size is a choice between 2, 4, 8, and 16.\n",
    "\n",
    "At each trial, Ray Tune will now randomly sample a combination of parameters from these\n",
    "search spaces. It will then train a number of models in parallel and find the best\n",
    "performing one among these. We also use the ``ASHAScheduler`` which will terminate bad\n",
    "performing trials early.\n",
    "\n",
    "We wrap the ``train_cifar`` function with ``functools.partial`` to set the constant\n",
    "``data_dir`` parameter. We can also tell Ray Tune what resources should be\n",
    "available for each trial:\n",
    "\n",
    "```python\n",
    "gpus_per_trial = 2\n",
    "# ...\n",
    "result = tune.run(\n",
    "    partial(train_cifar, data_dir=data_dir),\n",
    "    resources_per_trial={\"cpu\": 8, \"gpu\": gpus_per_trial},\n",
    "    config=config,\n",
    "    num_samples=num_samples,\n",
    "    scheduler=scheduler,\n",
    "    progress_reporter=reporter,\n",
    "    checkpoint_at_end=True)\n",
    "```\n",
    "You can specify the number of CPUs, which are then available e.g.\n",
    "to increase the ``num_workers`` of the PyTorch ``DataLoader`` instances. The selected\n",
    "number of GPUs are made visible to PyTorch in each trial. Trials do not have access to\n",
    "GPUs that haven't been requested for them - so you don't have to care about two trials\n",
    "using the same set of resources.\n",
    "\n",
    "Here we can also specify fractional GPUs, so something like ``gpus_per_trial=0.5`` is\n",
    "completely valid. The trials will then share GPUs among each other.\n",
    "You just have to make sure that the models still fit in the GPU memory.\n",
    "\n",
    "After training the models, we will find the best performing one and load the trained\n",
    "network from the checkpoint file. We then obtain the test set accuracy and report\n",
    "everything by printing.\n",
    "\n",
    "The full main function looks like this:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(655, 649)\n",
      "(655,)\n",
      "CPU has 56 cores on this device.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-06 02:36:30,926\tINFO utils.py:616 -- Detected RAY_USE_MULTIPROCESSING_CPU_COUNT=1: Using multiprocessing.cpu_count() to detect the number of CPUs. This may be inconsistent when used inside docker. To correctly detect CPUs, unset the env var: `RAY_USE_MULTIPROCESSING_CPU_COUNT`.\n",
      "2023-02-06 02:36:31,111\tINFO worker.py:1529 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py:610: DeprecationWarning: `checkpoint_dir` in `func(config, checkpoint_dir)` is being deprecated. To save and load checkpoint in trainable functions, please use the `ray.air.session` API:\n",
      "\n",
      "from ray.air import session\n",
      "\n",
      "def train(config):\n",
      "    # ...\n",
      "    session.report({\"metric\": metric}, checkpoint=checkpoint)\n",
      "\n",
      "For more information please see https://docs.ray.io/en/master/tune/api_docs/trainable.html\n",
      "\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Status ==\n",
      "Current time: 2023-02-06 02:36:43 (running for 00:00:02.99)\n",
      "Memory usage on this node: 33.2/125.8 GiB \n",
      "Using AsyncHyperBand: num_stopped=0\n",
      "Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None\n",
      "Resources requested: 56.0/56 CPUs, 1.0/1 GPUs, 0.0/7.94 GiB heap, 0.0/3.97 GiB objects (0.0/1.0 accelerator_type:G)\n",
      "Result logdir: /root/ray_results/train_2023-02-06_02-36-40\n",
      "Number of trials: 61/10800 (60 PENDING, 1 RUNNING)\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "| Trial name        | status   | loc               |   cnn_depth |   EPOCHS |    lr |   momentum | optimizer   | activation_function   |\n",
      "|-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------|\n",
      "| train_08077_00000 | RUNNING  | 172.17.0.7:181349 |           1 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00001 | PENDING  |                   |           1 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00002 | PENDING  |                   |           1 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00003 | PENDING  |                   |           2 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00004 | PENDING  |                   |           2 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00005 | PENDING  |                   |           2 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00006 | PENDING  |                   |           3 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00007 | PENDING  |                   |           3 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00008 | PENDING  |                   |           3 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00009 | PENDING  |                   |           4 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00010 | PENDING  |                   |           4 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00011 | PENDING  |                   |           4 |       30 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00012 | PENDING  |                   |           5 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00013 | PENDING  |                   |           5 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00014 | PENDING  |                   |           5 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00015 | PENDING  |                   |           6 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00016 | PENDING  |                   |           6 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00017 | PENDING  |                   |           6 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00018 | PENDING  |                   |           7 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00019 | PENDING  |                   |           7 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "... 41 more trials not shown (41 PENDING)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-06 02:36:52,400\tERROR trial_runner.py:1088 -- Trial train_08077_00000: Error processing event.\n",
      "ray.exceptions.RayTaskError(NameError): \u001b[36mray::ImplicitFunc.train()\u001b[39m (pid=181349, ip=172.17.0.7, repr=func)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/trainable.py\", line 367, in train\n",
      "    raise skipped from exception_cause(skipped)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 335, in entrypoint\n",
      "    return self._trainable_func(\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 652, in _trainable_func\n",
      "    output = fn()\n",
      "  File \"/tmp/ipykernel_176735/1430548902.py\", line 147, in train\n",
      "NameError: name 'trainset' is not defined\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"trialProgress\">\n",
       "  <h3>Trial Progress</h3>\n",
       "  <table>\n",
       "<thead>\n",
       "<tr><th>Trial name       </th><th>date               </th><th>experiment_id                   </th><th>hostname                            </th><th>node_ip   </th><th style=\"text-align: right;\">   pid</th><th style=\"text-align: right;\">  timestamp</th><th style=\"text-align: right;\">   trial_id</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>train_08077_00000</td><td>2023-02-06_02-36-49</td><td>99ec4d207e7442989696396fdbf61931</td><td>autodl-container-7e4011b6ac-94b7816d</td><td>172.17.0.7</td><td style=\"text-align: right;\">181349</td><td style=\"text-align: right;\"> 1675622209</td><td style=\"text-align: right;\">08077_00000</td></tr>\n",
       "<tr><td>train_08077_00001</td><td>2023-02-06_02-36-58</td><td>28ddb41f9f1446659d738189fa7bc8df</td><td>autodl-container-7e4011b6ac-94b7816d</td><td>172.17.0.7</td><td style=\"text-align: right;\">181502</td><td style=\"text-align: right;\"> 1675622218</td><td style=\"text-align: right;\">08077_00001</td></tr>\n",
       "<tr><td>train_08077_00002</td><td>2023-02-06_02-37-06</td><td>13fa543a800740a5bc3ccc1131dc355b</td><td>autodl-container-7e4011b6ac-94b7816d</td><td>172.17.0.7</td><td style=\"text-align: right;\">181656</td><td style=\"text-align: right;\"> 1675622226</td><td style=\"text-align: right;\">08077_00002</td></tr>\n",
       "<tr><td>train_08077_00003</td><td>2023-02-06_02-37-15</td><td>84be26247c5845418ee9904fad9217e9</td><td>autodl-container-7e4011b6ac-94b7816d</td><td>172.17.0.7</td><td style=\"text-align: right;\">181808</td><td style=\"text-align: right;\"> 1675622235</td><td style=\"text-align: right;\">08077_00003</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "</div>\n",
       "<style>\n",
       ".trialProgress {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".trialProgress h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".trialProgress td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Status ==\n",
      "Current time: 2023-02-06 02:36:52 (running for 00:00:11.66)\n",
      "Memory usage on this node: 33.3/125.8 GiB \n",
      "Using AsyncHyperBand: num_stopped=0\n",
      "Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None\n",
      "Resources requested: 0/56 CPUs, 0/1 GPUs, 0.0/7.94 GiB heap, 0.0/3.97 GiB objects (0.0/1.0 accelerator_type:G)\n",
      "Result logdir: /root/ray_results/train_2023-02-06_02-36-40\n",
      "Number of trials: 62/10800 (1 ERROR, 61 PENDING)\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "| Trial name        | status   | loc               |   cnn_depth |   EPOCHS |    lr |   momentum | optimizer   | activation_function   |\n",
      "|-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------|\n",
      "| train_08077_00001 | PENDING  |                   |           1 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00002 | PENDING  |                   |           1 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00003 | PENDING  |                   |           2 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00004 | PENDING  |                   |           2 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00005 | PENDING  |                   |           2 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00006 | PENDING  |                   |           3 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00007 | PENDING  |                   |           3 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00008 | PENDING  |                   |           3 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00009 | PENDING  |                   |           4 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00010 | PENDING  |                   |           4 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00011 | PENDING  |                   |           4 |       30 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00012 | PENDING  |                   |           5 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00013 | PENDING  |                   |           5 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00014 | PENDING  |                   |           5 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00015 | PENDING  |                   |           6 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00016 | PENDING  |                   |           6 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00017 | PENDING  |                   |           6 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00018 | PENDING  |                   |           7 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00019 | PENDING  |                   |           7 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00000 | ERROR    | 172.17.0.7:181349 |           1 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "... 42 more trials not shown (42 PENDING)\n",
      "Number of errored trials: 1\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "| Trial name        |   # failures | error file                                                                                                                                                                             |\n",
      "|-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| train_08077_00000 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00000_0_EPOCHS=10,activation_function=lrelu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-43/error.txt |\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n",
      "== Status ==\n",
      "Current time: 2023-02-06 02:36:57 (running for 00:00:16.93)\n",
      "Memory usage on this node: 32.5/125.8 GiB \n",
      "Using AsyncHyperBand: num_stopped=0\n",
      "Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None\n",
      "Resources requested: 56.0/56 CPUs, 1.0/1 GPUs, 0.0/7.94 GiB heap, 0.0/3.97 GiB objects (0.0/1.0 accelerator_type:G)\n",
      "Result logdir: /root/ray_results/train_2023-02-06_02-36-40\n",
      "Number of trials: 63/10800 (1 ERROR, 61 PENDING, 1 RUNNING)\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "| Trial name        | status   | loc               |   cnn_depth |   EPOCHS |    lr |   momentum | optimizer   | activation_function   |\n",
      "|-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------|\n",
      "| train_08077_00001 | RUNNING  | 172.17.0.7:181502 |           1 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00002 | PENDING  |                   |           1 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00003 | PENDING  |                   |           2 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00004 | PENDING  |                   |           2 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00005 | PENDING  |                   |           2 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00006 | PENDING  |                   |           3 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00007 | PENDING  |                   |           3 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00008 | PENDING  |                   |           3 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00009 | PENDING  |                   |           4 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00010 | PENDING  |                   |           4 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00011 | PENDING  |                   |           4 |       30 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00012 | PENDING  |                   |           5 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00013 | PENDING  |                   |           5 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00014 | PENDING  |                   |           5 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00015 | PENDING  |                   |           6 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00016 | PENDING  |                   |           6 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00017 | PENDING  |                   |           6 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00018 | PENDING  |                   |           7 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00019 | PENDING  |                   |           7 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00000 | ERROR    | 172.17.0.7:181349 |           1 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "... 43 more trials not shown (43 PENDING)\n",
      "Number of errored trials: 1\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "| Trial name        |   # failures | error file                                                                                                                                                                             |\n",
      "|-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| train_08077_00000 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00000_0_EPOCHS=10,activation_function=lrelu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-43/error.txt |\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-06 02:37:01,279\tERROR trial_runner.py:1088 -- Trial train_08077_00001: Error processing event.\n",
      "ray.exceptions.RayTaskError(NameError): \u001b[36mray::ImplicitFunc.train()\u001b[39m (pid=181502, ip=172.17.0.7, repr=func)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/trainable.py\", line 367, in train\n",
      "    raise skipped from exception_cause(skipped)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 335, in entrypoint\n",
      "    return self._trainable_func(\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 652, in _trainable_func\n",
      "    output = fn()\n",
      "  File \"/tmp/ipykernel_176735/1430548902.py\", line 147, in train\n",
      "NameError: name 'trainset' is not defined\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Status ==\n",
      "Current time: 2023-02-06 02:37:06 (running for 00:00:25.83)\n",
      "Memory usage on this node: 31.0/125.8 GiB \n",
      "Using AsyncHyperBand: num_stopped=0\n",
      "Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None\n",
      "Resources requested: 56.0/56 CPUs, 1.0/1 GPUs, 0.0/7.94 GiB heap, 0.0/3.97 GiB objects (0.0/1.0 accelerator_type:G)\n",
      "Result logdir: /root/ray_results/train_2023-02-06_02-36-40\n",
      "Number of trials: 64/10800 (2 ERROR, 61 PENDING, 1 RUNNING)\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "| Trial name        | status   | loc               |   cnn_depth |   EPOCHS |    lr |   momentum | optimizer   | activation_function   |\n",
      "|-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------|\n",
      "| train_08077_00002 | RUNNING  | 172.17.0.7:181656 |           1 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00003 | PENDING  |                   |           2 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00004 | PENDING  |                   |           2 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00005 | PENDING  |                   |           2 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00006 | PENDING  |                   |           3 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00007 | PENDING  |                   |           3 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00008 | PENDING  |                   |           3 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00009 | PENDING  |                   |           4 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00010 | PENDING  |                   |           4 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00011 | PENDING  |                   |           4 |       30 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00012 | PENDING  |                   |           5 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00013 | PENDING  |                   |           5 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00014 | PENDING  |                   |           5 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00015 | PENDING  |                   |           6 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00016 | PENDING  |                   |           6 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00017 | PENDING  |                   |           6 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00018 | PENDING  |                   |           7 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00019 | PENDING  |                   |           7 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00000 | ERROR    | 172.17.0.7:181349 |           1 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00001 | ERROR    | 172.17.0.7:181502 |           1 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "... 44 more trials not shown (44 PENDING)\n",
      "Number of errored trials: 2\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "| Trial name        |   # failures | error file                                                                                                                                                                             |\n",
      "|-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| train_08077_00000 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00000_0_EPOCHS=10,activation_function=lrelu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-43/error.txt |\n",
      "| train_08077_00001 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00001_1_EPOCHS=20,activation_function=relu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-52/error.txt  |\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-06 02:37:09,241\tERROR trial_runner.py:1088 -- Trial train_08077_00002: Error processing event.\n",
      "ray.exceptions.RayTaskError(NameError): \u001b[36mray::ImplicitFunc.train()\u001b[39m (pid=181656, ip=172.17.0.7, repr=func)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/trainable.py\", line 367, in train\n",
      "    raise skipped from exception_cause(skipped)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 335, in entrypoint\n",
      "    return self._trainable_func(\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 652, in _trainable_func\n",
      "    output = fn()\n",
      "  File \"/tmp/ipykernel_176735/1430548902.py\", line 147, in train\n",
      "NameError: name 'trainset' is not defined\n",
      "2023-02-06 02:37:15,289\tERROR trial_runner.py:1088 -- Trial train_08077_00003: Error processing event.\n",
      "ray.exceptions.RayTaskError(NameError): \u001b[36mray::ImplicitFunc.train()\u001b[39m (pid=181808, ip=172.17.0.7, repr=func)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/trainable.py\", line 367, in train\n",
      "    raise skipped from exception_cause(skipped)\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 335, in entrypoint\n",
      "    return self._trainable_func(\n",
      "  File \"/root/miniconda3/envs/tf/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py\", line 652, in _trainable_func\n",
      "    output = fn()\n",
      "  File \"/tmp/ipykernel_176735/1430548902.py\", line 99, in train\n",
      "  File \"/tmp/ipykernel_176735/987737040.py\", line 29, in __init__\n",
      "NameError: name 'ac_dict' is not defined\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Status ==\n",
      "Current time: 2023-02-06 02:37:14 (running for 00:00:33.83)\n",
      "Memory usage on this node: 32.0/125.8 GiB \n",
      "Using AsyncHyperBand: num_stopped=0\n",
      "Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None\n",
      "Resources requested: 56.0/56 CPUs, 1.0/1 GPUs, 0.0/7.94 GiB heap, 0.0/3.97 GiB objects (0.0/1.0 accelerator_type:G)\n",
      "Result logdir: /root/ray_results/train_2023-02-06_02-36-40\n",
      "Number of trials: 65/10800 (3 ERROR, 61 PENDING, 1 RUNNING)\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "| Trial name        | status   | loc               |   cnn_depth |   EPOCHS |    lr |   momentum | optimizer   | activation_function   |\n",
      "|-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------|\n",
      "| train_08077_00003 | RUNNING  | 172.17.0.7:181808 |           2 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00004 | PENDING  |                   |           2 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00005 | PENDING  |                   |           2 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00006 | PENDING  |                   |           3 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00007 | PENDING  |                   |           3 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00008 | PENDING  |                   |           3 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00009 | PENDING  |                   |           4 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00010 | PENDING  |                   |           4 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00011 | PENDING  |                   |           4 |       30 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00012 | PENDING  |                   |           5 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00013 | PENDING  |                   |           5 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00014 | PENDING  |                   |           5 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00015 | PENDING  |                   |           6 |       10 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00016 | PENDING  |                   |           6 |       20 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00017 | PENDING  |                   |           6 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00018 | PENDING  |                   |           7 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00019 | PENDING  |                   |           7 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00000 | ERROR    | 172.17.0.7:181349 |           1 |       10 | 0.001 |        0.5 | Adam        | lrelu                 |\n",
      "| train_08077_00001 | ERROR    | 172.17.0.7:181502 |           1 |       20 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "| train_08077_00002 | ERROR    | 172.17.0.7:181656 |           1 |       30 | 0.001 |        0.5 | Adam        | relu                  |\n",
      "+-------------------+----------+-------------------+-------------+----------+-------+------------+-------------+-----------------------+\n",
      "... 45 more trials not shown (45 PENDING)\n",
      "Number of errored trials: 3\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "| Trial name        |   # failures | error file                                                                                                                                                                             |\n",
      "|-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| train_08077_00000 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00000_0_EPOCHS=10,activation_function=lrelu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-43/error.txt |\n",
      "| train_08077_00001 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00001_1_EPOCHS=20,activation_function=relu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-36-52/error.txt  |\n",
      "| train_08077_00002 |            1 | /root/ray_results/train_2023-02-06_02-36-40/train_08077_00002_2_EPOCHS=30,activation_function=relu,cnn_depth=1,lr=0.0010,momentum=0.5000,optimizer=Adam_2023-02-06_02-37-01/error.txt  |\n",
      "+-------------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):\n",
    "#     data_dir = os.path.abspath(\"./data\")\n",
    "#     load_data(data_dir)\n",
    "#     config = {\n",
    "#         \"l1\": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),\n",
    "#         \"l2\": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),\n",
    "#         \"lr\": tune.loguniform(1e-4, 1e-1),\n",
    "#         \"batch_size\": tune.choice([2, 4, 8, 16])\n",
    "#     }\n",
    "#     scheduler = ASHAScheduler(\n",
    "#         metric=\"loss\",\n",
    "#         mode=\"min\",\n",
    "#         max_t=max_num_epochs,\n",
    "#         grace_period=1,\n",
    "#         reduction_factor=2)\n",
    "#     reporter = CLIReporter(\n",
    "#         # parameter_columns=[\"l1\", \"l2\", \"lr\", \"batch_size\"],\n",
    "#         metric_columns=[\"loss\", \"accuracy\", \"training_iteration\"])\n",
    "#     result = tune.run(\n",
    "#         partial(train_cifar, data_dir=data_dir),\n",
    "#         resources_per_trial={\"cpu\": 2, \"gpu\": gpus_per_trial},\n",
    "#         config=config,\n",
    "#         num_samples=num_samples,\n",
    "#         scheduler=scheduler,\n",
    "#         progress_reporter=reporter)\n",
    "#\n",
    "#     best_trial = result.get_best_trial(\"loss\", \"min\", \"last\")\n",
    "#     print(\"Best trial config: {}\".format(best_trial.config))\n",
    "#     print(\"Best trial final validation loss: {}\".format(\n",
    "#         best_trial.last_result[\"loss\"]))\n",
    "#     print(\"Best trial final validation accuracy: {}\".format(\n",
    "#         best_trial.last_result[\"accuracy\"]))\n",
    "#\n",
    "#     best_trained_model = Net(best_trial.config[\"l1\"], best_trial.config[\"l2\"])\n",
    "#     device = \"cpu\"\n",
    "#     if torch.cuda.is_available():\n",
    "#         device = \"cuda:0\"\n",
    "#         if gpus_per_trial > 1:\n",
    "#             best_trained_model = nn.DataParallel(best_trained_model)\n",
    "#     best_trained_model.to(device)\n",
    "#\n",
    "#     best_checkpoint_dir = best_trial.checkpoint.value\n",
    "#     model_state, optimizer_state = torch.load(os.path.join(\n",
    "#         best_checkpoint_dir, \"checkpoint\"))\n",
    "#     best_trained_model.load_state_dict(model_state)\n",
    "#\n",
    "#     test_acc = test_accuracy(best_trained_model, device)\n",
    "#     print(\"Best trial test set accuracy: {}\".format(test_acc))\n",
    "#\n",
    "#\n",
    "# if __name__ == \"__main__\":\n",
    "#     # You can change the number of GPUs per trial here:\n",
    "#     main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)\n",
    "\n",
    "def autoTuneMain(num_samples=10, max_num_epochs=10, gpus_per_trial=1, data=None, label=None,\n",
    "                 ProcessMethods=None, FslecetedMethods=None, SetSplitMethods=None, model=None,\n",
    "                 cnn_depth=None, loss=None, config=None, max_cpu_cores=1):\n",
    "    # load data\n",
    "    data_wrapUp = load_data(data=data, label=label, \n",
    "                            ProcessMethods=ProcessMethods, \n",
    "                            FslecetedMethods=FslecetedMethods,\n",
    "                            SetSplitMethods=SetSplitMethods)\n",
    "\n",
    "    scheduler = ASHAScheduler(\n",
    "        metric=\"loss\",\n",
    "        mode=\"min\",\n",
    "        max_t=max_num_epochs,\n",
    "        grace_period=1,\n",
    "        reduction_factor=2)\n",
    "\n",
    "    reporter = CLIReporter(\n",
    "        parameter_columns=[STRING_CONFIG.cnn_depth, STRING_CONFIG.epochs, STRING_CONFIG.lr, STRING_CONFIG.momentum,\n",
    "                           STRING_CONFIG.optimizer, STRING_CONFIG.acti_func],\n",
    "        metric_columns=[\"loss\", \"accuracy\", \"training_iteration\"])\n",
    "\n",
    "    # 使用字典映射调用 loss函数\n",
    "    LOSS_DICT = {\n",
    "        STRING_CONFIG.MSE: nn.MSELoss(),\n",
    "        STRING_CONFIG.L1: nn.L1Loss(),\n",
    "        STRING_CONFIG.CrossEntropy: nn.CrossEntropyLoss(ignore_index=-100),\n",
    "        STRING_CONFIG.Poisson: nn.PoissonNLLLoss(log_input=True, full=False, eps=1e-08),\n",
    "        # 'KLDiv': nn.KLDivLoss(reduction='batchmean'),\n",
    "    }\n",
    "\n",
    "    result = tune.run(\n",
    "        partial(train, data_wrapUp=data_wrapUp),\n",
    "        resources_per_trial={\"cpu\": max_cpu_cores, \"gpu\": gpus_per_trial},\n",
    "        config=config,\n",
    "        num_samples=num_samples,\n",
    "        scheduler=scheduler,\n",
    "        progress_reporter=reporter)\n",
    "\n",
    "    best_trial = result.get_best_trial(\"loss\", \"min\", \"last\")\n",
    "    print(\"Best trial config: {}\".format(best_trial.config))\n",
    "    print(\"Best trial final validation loss: {}\".format(\n",
    "        best_trial.last_result[\"loss\"]))\n",
    "    print(\"Best trial final validation accuracy: {}\".format(\n",
    "        best_trial.last_result[\"accuracy\"]))\n",
    "\n",
    "    # ============================= 自动超参完成后 ====================================\n",
    "    best_trained_model = Net(best_trial.config[STRING_CONFIG.acti_func], best_trial.config[STRING_CONFIG.cnn_depth])\n",
    "    device = \"cpu\"\n",
    "    if torch.cuda.is_available():\n",
    "        device = \"cuda:0\"\n",
    "        if gpus_per_trial > 1:\n",
    "            best_trained_model = nn.DataParallel(best_trained_model)\n",
    "    best_trained_model.to(device)\n",
    "\n",
    "    best_checkpoint_dir = best_trial.checkpoint.value\n",
    "    model_state, optimizer_state = torch.load(os.path.join(\n",
    "        best_checkpoint_dir, \"checkpoint\"))\n",
    "    best_trained_model.load_state_dict(model_state)\n",
    "\n",
    "    test_acc = test_accuracy(best_trained_model, device, data_wrapUp=data_wrapUp)\n",
    "    print(\"Best trial test set accuracy: {}\".format(test_acc))\n",
    "    # ============================= 自动超参完成后 ====================================\n",
    "    \n",
    "if __name__ == '__main__':\n",
    "    # 载入原始数据并可视化\n",
    "    data2, label2 = LoadNirtest('Rgs')\n",
    "\n",
    "    # 是否使用自动超参功能\n",
    "    is_autoTune = True\n",
    "    if is_autoTune:\n",
    "        # 获得本设备CPU最大核心数量。\n",
    "        #   注意：如果出现可用内存空间不足，请自行调整`max_cpu_cores`以减少CPU核心数的使用！\n",
    "        max_cpu_cores = multiprocessing.cpu_count()\n",
    "        print(\"CPU has \" + str(max_cpu_cores) + \" cores on this device.\")\n",
    "\n",
    "        # 设置环境变量，解决Ray Tune因为CPU检测不准确而给出的warning。\n",
    "        os.environ['RAY_USE_MULTIPROCESSING_CPU_COUNT'] = '1'\n",
    "        # 设置超参数搜索空间\n",
    "        search_space = {\n",
    "            STRING_CONFIG.cnn_depth: tune.grid_search([1, 2, 3, 4, 5, 6, 7, 8]),\n",
    "            STRING_CONFIG.epochs: tune.grid_search([10, 20, 30]),  # 具体EPOCHS搜参空间请根据模型的收敛情况判断而定\n",
    "            STRING_CONFIG.lr: tune.grid_search([.001, .01, .1]),\n",
    "            STRING_CONFIG.momentum: tune.grid_search([.5, .9, .99]),\n",
    "            STRING_CONFIG.optimizer: tune.grid_search([\"Adam\", \"SGD\", \"Adadelta\", \"RMSprop\", \"Adamax\"]),\n",
    "            STRING_CONFIG.acti_func: tune.choice([\"relu\", \"lrelu\"])\n",
    "        }\n",
    "        autoTuneMain(num_samples=10, max_num_epochs=10, gpus_per_trial=1,\n",
    "                     data=data2, label=label2, ProcessMethods=\"MMS\", FslecetedMethods=\"None\",\n",
    "                     SetSplitMethods=\"random\", model=\"CNN_vgg\", cnn_depth=5,\n",
    "                     loss=STRING_CONFIG.MSE, config=search_space, max_cpu_cores=max_cpu_cores)\n",
    "    else:\n",
    "        # 光谱定量分析演示\n",
    "        # 示意1: 预处理算法:MSC , 波长筛选算法: Uve, 数据集划分:KS, 定性分量模型: SVR\n",
    "        # 这里我改了参数，10,'relu',5,'MSE','Adam'，10 是epoch，relu是激活函数，可以选。5是CNN有5层，MSE是损失函数，Adam是优化，这些可以到CNN.p文件看一下有什么选择。\n",
    "        SpectralQuantitativeAnalysis(data=data2, label=label2,\n",
    "                                     ProcessMethods=\"MMS\", FslecetedMethods=\"None\",\n",
    "                                     SetSplitMethods=\"random\",\n",
    "                                     model=\"CNN_vgg\", EPOCH=3, acti='relu',\n",
    "                                     cnn_depth=5, loss='MSE', optim='SGD',\n",
    "                                     is_autoTune=False,\n",
    "                                     autoHyperConfig=search_space)\n",
    "        # RMSE, R2, MAE = SpectralQuantitativeAnalysis(data=data2, label=label2, ProcessMethods=\"MMS\",\n",
    "        #                                              FslecetedMethods=\"None\", SetSplitMethods=\"random\",\n",
    "        #                                              model=\"CNN_vgg\", EPOCH=3, acti='relu', c_num=5,\n",
    "        #                                              loss='MSE', optim='Adam')\n",
    "\n",
    "        # print(\"The RMSE:{} R2:{}, MAE:{} of result!\".format(RMSE, R2, MAE))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "If you run the code, an example output could look like this:\n",
    "\n",
    "::\n",
    "\n",
    "    Number of trials: 10 (10 TERMINATED)\n",
    "    +-----+------+------+-------------+--------------+---------+------------+--------------------+\n",
    "    | ... |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |\n",
    "    |-----+------+------+-------------+--------------+---------+------------+--------------------|\n",
    "    | ... |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |\n",
    "    | ... |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |\n",
    "    | ... |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |\n",
    "    | ... |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |\n",
    "    | ... |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |\n",
    "    | ... |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |\n",
    "    | ... |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |\n",
    "    | ... |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |\n",
    "    | ... |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |\n",
    "    | ... |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |\n",
    "    +-----+------+------+-------------+--------------+---------+------------+--------------------+\n",
    "\n",
    "\n",
    "    Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}\n",
    "    Best trial final validation loss: 1.181501\n",
    "    Best trial final validation accuracy: 0.5836\n",
    "    Best trial test set accuracy: 0.5806\n",
    "\n",
    "Most trials have been stopped early in order to avoid wasting resources.\n",
    "The best performing trial achieved a validation accuracy of about 58%, which could\n",
    "be confirmed on the test set.\n",
    "\n",
    "So that's it! You can now tune the parameters of your PyTorch models.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf",
   "language": "python",
   "name": "tf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
